<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<HEAD>
	<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=iso-8859-1">
	<TITLE>OSBF-Lua</TITLE>
	<META NAME="GENERATOR" CONTENT="OpenOffice.org 2.0  (Linux)">
	<META NAME="CREATED" CONTENT="20050304;23093600">
	<META NAME="CHANGEDBY" CONTENT="Fidelis Assis">
	<META NAME="CHANGED" CONTENT="20060103;22242800">
</HEAD>
<BODY LANG="pt-BR" BGCOLOR="#ffffff" DIR="LTR">
<HR>
<CENTER>
	<TABLE BORDER=0 CELLPADDING=2 CELLSPACING=2>
		<TR>
			<TD></TD>
		</TR>
		<TR>
			<TD>
				<P ALIGN=CENTER><B><FONT SIZE=4>OSBF-Lua Reference Manual</FONT></B>
								</P>
			</TD>
		</TR>
		<TR>
			<TD VALIGN=TOP>
				<P ALIGN=CENTER>Text classification library for the <A HREF="http://www.lua.org/">Lua</A>
				programming language 
				</P>
			</TD>
		</TR>
	</TABLE>
</CENTER>
<P ALIGN=CENTER><FONT SIZE=2><A HREF="index.html">home</A> &middot;
<A HREF="#introduction">introduction</A> &middot; <A HREF="#reference">reference</A>
&middot; <A HREF="#examples">examples</A> </FONT>
</P>
<HR>
<H2><A NAME="introduction"></A>Introduction</H2>
<P>OSBF-Lua (Orthogonal Sparse Bigrams with confidence Factor) is a C
module for text classification written for <A HREF="http://www.lua.org/">Lua</A>.
It is a port of the OSBF classifier implemented in CRM114,
<A HREF="http://crm114.sf.net/"><U>http://crm114.sf.net.</U></A> It
borrows many good ideas from Bill Yerazunis' CRM114, like the
databases basic structure and the Bayesian chain implementation. The
OSBF algorithm is a typical Bayesian classifier but enhanced with the
OSB (Orthogonal Sparse Bigrams) feature extraction technique and an
ad hoc Confidence Factor (or &ldquo;voodoo&rdquo;), for automatic
reduction of the less significant features impact on the
classification &ndash; noise reduction. The final result is a very
fast and accurate classifier. It was developed focused on 2 classes,
SPAM and NON-SPAM, so the performance with more than 2 classes may
not be the same.</P>
<P>OSBF-Lua is free software and is released under the GPL version 2.
You can get a copy of the license at <A HREF="http://www.fsf.org/licensing/licenses/gpl.txt">GPL</A>.
This distribuition includes a copy of the license in the file
gpl.txt.</P>
<H2><A NAME="reference"></A>Reference</H2>
<P>OSBF-Lua offers the following functions: 
</P>
<UL>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="create_db"></A><TT><B><FONT FACE="Bitstream Vera Sans Mono">osbf.create_db(classes,
	num_buckets)</FONT></B></TT></P>
	<P STYLE="margin-bottom: 0cm">Creates the single class databases
	specified in the table <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">classes</FONT></FONT></TT>,
	with <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">num_buckets</FONT></FONT></TT>
	buckets each. 
	</P>
	<P STYLE="margin-bottom: 0cm"><B><FONT FACE="Bitstream Vera Sans Mono">osbf.create</FONT></B>
	returns the number of single class databases created or <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></TT>
	plus an error message.</P>
	<P STYLE="margin-bottom: 0cm"><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">Ex:
	</FONT></FONT><FONT SIZE=2><FONT FACE="Bitstream Vera Sans Mono">osbf.create_db({&ldquo;nonspam.cfc&rdquo;,
	&ldquo;spam.cfc&rdquo;}, 94321)</FONT></FONT></TT></P>
	<P STYLE="margin-bottom: 0cm"></P>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="remove_db"></A><TT><B><FONT FACE="Bitstream Vera Sans Mono">osbf.remove_db
	(classes)</FONT></B></TT><BR>Removes all single class databases
	specified in the table <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">classes</FONT></FONT></TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">.
	</FONT></FONT><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">classes</FONT></FONT></TT>
	is the same as in <TT><B><FONT FACE="Bitstream Vera Sans Mono">osbf.create_db</FONT></B></TT>.</P>
	<P STYLE="margin-bottom: 0cm"><B><FONT FACE="Bitstream Vera Sans Mono">osbf.remove</FONT></B>
	returns <TT>true</TT> in case of success or <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></TT>
	plus an error message.</P>
	<P STYLE="margin-bottom: 0cm"><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">Ex:
	</FONT></FONT><FONT SIZE=2><FONT FACE="Bitstream Vera Sans Mono">osbf.remove_db({&ldquo;nonspam.cfc&rdquo;,
	&ldquo;spam.cfc&rdquo;})</FONT></FONT></TT></P>
	<P STYLE="margin-bottom: 0cm"></P>
	<LI><P><A NAME="classify"></A><TT><B><FONT FACE="Bitstream Vera Sans Mono">osbf.classify(text,
	dbset, flags, min_p_ratio)</FONT></B></TT><BR>Classifies the string
	<TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">text</FONT></FONT></TT>.
		</P>
</UL>
<P STYLE="margin-left: 1.27cm; margin-bottom: 0cm"><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">text</FONT></FONT></B><FONT FACE="Bitstream Vera Sans Mono">:
String with the text to be classified;</FONT></P>
<P STYLE="margin-left: 1.27cm; margin-bottom: 0cm"><BR>
</P>
<P STYLE="margin-left: 1.27cm; margin-bottom: 0cm"><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dbset</FONT></FONT></B>:
Lua table with the following structure: 
</P>
<P STYLE="margin-left: 1.27cm; margin-bottom: 0cm"><BR>
</P>
<UL>
	<PRE>    dbset = {
           classes    = {&quot;nonspam.cfc&quot;, &quot;spam.cfc&quot;},
           ncfs       = 1,
           delimiters = &quot;&quot; -- you can put additional token delimiters here
    }</PRE><P>
	<B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">ncfs</FONT></FONT><I>:</I></B>
	splits <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">classes</FONT></FONT><TT>
	in 2 subsets. The first subset is formed by the first <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">ncfs</FONT></FONT>
	class databases. The remainder databases will form the second
	subset. These 2 subsets define 2 composed classes. In the above
	example we have 2 composed classes formed by a single class database
	each. Another possibility, for instance, would be 2 composed classes
	formed by a pair of single class databases each: global and per
	user. Ex: </TT>
	</P>
	<PRE>    dbset = {
           classes = {&quot;globalnonspam.cfc&quot;, &quot;usernonspam.cfc&quot;, &quot;globalspam.cfc&quot;, &quot;userspam.cfc&quot;},
           ncfs = 2, -- 2 single classes in the first subset
           delimiters = &quot;&quot;
    }</PRE><P STYLE="margin-bottom: 0cm">
	<TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">flags</FONT></FONT></B>:
	Number with the flags to control classification. Set to 0 for normal
	use. Each bit of the number is a flag. For now, there's only one
	flag defined, the NO_VOODOO flag. That is, set flags to 1 to disable
	the voodoo formula. The NO_VOODOO flag is intended more for test
	purposes because disabling it normally lowers accuracy.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">min_p_ratio</FONT></FONT></B>:
	Number with the minimum feature probability ratio. The probability
	ratio of a feature is the ratio between the maximum and the minimum
	probabilities it has over the classes. Features with less than
	<FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">min_p_ratio</FONT></FONT>
	are not considered for classification. This parameter is optional.
	The<TT> default is 1, which means that all features are considered.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">delimiters</FONT></FONT>:
	String with extra token delimiters. The tokens are produced by the
	internal fixed pattern ([[:graph:]]+), or, in other words, by
	sequences of printable chars except tab, new line, vertical tab,
	form feed, carriage return, or space. If <TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">delimiters</FONT></FONT></TT>
	is not empty, its chars will be considered as extra token
	delimiters, like space, tab, new line, etc.</TT></P>
	<P><TT><BR><B>osbf.classify</B> returns 3 values, in the following
	order:</TT></P>
</UL>
<UL>
	<P STYLE="margin-bottom: 0cm"><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">.
	pR</FONT></FONT>: The log of the ratio between the probabilities of
	the first and second subset; </TT>
	</P>
	<P STYLE="margin-bottom: 0cm"><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">.
	p_array</FONT></FONT>: a Lua array with each single class
	probability; </TT>
	</P>
	<P><TT><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">.
	i_pmax</FONT></FONT>: index of the array to the single class with
	maximum probability; </TT>
	</P>
</UL>
<UL>
	<P><TT>In case of error, it returns 2 values: <CODE>nil</CODE> and
	an error message.</TT></P>
	<LI><P><A NAME="learn"></A><TT><TT><B>osbf.learn (text, dbset,
	class_index, flags)</B></TT> <BR>Learns the string <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">text</FONT></FONT>
	as belonging to the single class database indicated by the number
	<FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">class_index</FONT></FONT>
	in <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dbset.classes</FONT></FONT>.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">text</FONT></FONT></B>:
	string with the text to be learned;</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dbset</FONT></FONT></B>:
	table with the classes. Same structure as in <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">osbf.classify</FONT></FONT>;</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">class_index</FONT></FONT></B>:
	index to the single class, in db.classes, to be trained with <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">text</FONT></FONT>;</TT></P>
	<P><TT><B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">flags</FONT></FONT></B>:
	<TT>Number with the flags to control the learning operation. Set to
	0 for normal use. Each bit of the number is a flag. For now, there's
	only one flag defined, the NO_MICROGROOM flag. That is, set flags to
	1 to disable microgrooming. The <TT>NO_MICROGROOM</TT> flag is
	intended more for test purposes because the databases have fixed
	size and the pruning mechanism is necessary to guarantee space for
	new learnings.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>osbf.learn returns <I>true</I> in
	case of success or <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT>
	plus an error message in case of error.</TT></P>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="learn1"></A><TT><TT><B>osbf.config
	(options)</B></TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Configures internal parameters.
	This function is intended more for test purposes.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><TT><B>options: </B>table whose
	keys are the options to be set to their respective values.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><BR>The recognized options are:</TT></P>
	<UL>
		<LI><P STYLE="margin-bottom: 0cm"><TT><I>max_chain</I>: the max
		number of buckets allowed in a database chain. From that size on,
		the chain is pruned before inserting a new bucket;</TT></P>
		<LI><P STYLE="margin-bottom: 0cm"><TT><I>stop_after</I>: max number
		of buckets pruned in a chain;</TT></P>
		<LI><P STYLE="margin-bottom: 0cm"><TT><I>K1, K2, K3</I>: Constants
		used in the &ldquo;voodoo&rdquo; formula;</TT></P>
	</UL>
	<P STYLE="margin-bottom: 0cm"><TT><TT>Return the number of options
	set.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Ex: <FONT SIZE=2><FONT FACE="Bitstream Vera Sans Mono">osbf.config({max_chain
	= 50, stop_after = 100})</FONT></FONT></TT></P>
</UL>
<P STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="stats"></A><TT><TT><B>osbf.stats
	(dbfile)</B></TT> <BR>Returns an array with information and
	statistics of the specified <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">database</FONT></FONT>.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B>dbfile</B>: string with the
	database filename.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><BR>In case of error, it returns
	<CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></P>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="dump"></A><TT><B>osbf.dump
	(dbfile, csvfile)</B></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Creates <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">csvfile</FONT></FONT>,
	a dump of <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dbfile</FONT></FONT>
	in CSV format.</TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B>dbfile</B>:<TT> string with the
	database filename.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B>csvfile</B>: <TT>string with
	the csv filename.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>In case of error, it returns <CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></P>
</UL>
<P STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="restore"></A><TT><B>osbf.restore
	(dbfile, csvfile)</B></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Restores <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dbfile</FONT></FONT>
	from <FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">cvsfile</FONT></FONT>.
	<TT>In case of error, it returns <CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></TT></P>
</UL>
<P STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<P STYLE="margin-bottom: 0cm"><TT><B>dbfile</B>:<TT> string with the
	database filename.</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT><B>csvfile</B>: <TT>string with
	the csv filename</TT></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>In case of error, it returns <CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></P>
</UL>
<P STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="chdir"></A><TT><B>osbf.chdir
	(dir)</B></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Change the current working dir to
	<B><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">dir</FONT></FONT></B>.</TT></P>
</UL>
<P STYLE="margin-bottom: 0cm"><BR>
</P>
<UL>
	<P STYLE="margin-bottom: 0cm"><TT><B>dir</B>:<TT> string with the
	dirname.<TT> <TT>In case of error, it returns <CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></TT></TT></TT></P>
</UL>
<UL>
	<LI><P STYLE="margin-bottom: 0cm"><A NAME="getdir"></A><TT><B>osbf.getdir
	()</B></TT></P>
	<P STYLE="margin-bottom: 0cm"><TT>Returns the current working dir.
	<TT>In case of error, it returns <CODE><FONT SIZE=3 STYLE="font-size: 13pt"><FONT FACE="Courier, monospace">nil</FONT></FONT></CODE>
	plus an error message.</TT></TT></P>
</UL>
<H2><A NAME="examples"></A><TT>Examples</TT></H2>
<PRE>create_databases.lua:

-- Script for creating the databases

require &quot;osbf&quot;

-- class databases to be created
dbset = { classes = {&quot;nonspam.cfc&quot;, &quot;spam.cfc&quot;} }

-- number of buckets in each database
num_buckets = 94321

-- remove previous databases with the same name
osbf.remove_db(dbset.classes)

-- create new, empty databases
osbf.create_db(dbset.classes, num_buckets)

----------------------------------------------------------------------------


classify.lua:

-- Script for classifying a message read from stdin

require &quot;osbf&quot;

dbset = {
          classes = {&quot;nonspam.cfc&quot;, &quot;spam.cfc&quot;},
          ncfs = 1,
          delimiters = &quot;&quot;
}
classify_flags = 0

-- read entire message into var &quot;text&quot;
text = io.read(&quot;*all&quot;)

pR, p_array, i_pmax = osbf.classify(text, dbset, classify_flags)

if (pR == nil) then
   print(p_array)  -- in case of error, p_array contains the error message
else
   io.write(string.format(&quot;The message score is %f - &quot;, pR))
   if (pR &gt;= 0) then
     io.write(&quot;HAM\n&quot;)
   else
     io.write(&quot;SPAM\n&quot;)
   end
end


See more examples of the use of the osbf module in the spamfilter dir.
In special, take a look at the toer.lua script, which is a very fast
way of preparing your databases using a previously classified corpus
with your ham and spam messages.

----------------------------------------------------------------------------</PRE><P ALIGN=CENTER>
<TT><FONT SIZE=2><A HREF="index.html">home</A> &middot; <A HREF="#introduction">introduction</A>
&middot; <A HREF="#reference">reference</A> &middot; <A HREF="#examples">examples</A>
</FONT></TT>
</P>
<HR>
</BODY>
</HTML>